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EXECUTIVE  SUMMARY 


For  five  years  the  Office  of  the  Secretary  of  Defense  (OSD)  has  funded  the  Engineering  and 
Manufacturing  Development  (EMD)  Performance  Trends  Research  Project  at  the  Defense  Systems 
Management  College  (DSMC).  This  research  analyzes  the  cost,  schedule  and  performance  success 
of  Acquisition  Category  I  programs  during  the  EMD  phase  of  development.  This  Executive  Sum¬ 
mary  contains  the  major  conclusions  and  recommendations  of  the  research  effort  to  date.  Support¬ 
ing  rationale  and  details  are  contained  in  subsequent  sections  of  this  report. 

Major  Conclusions 

1.  The  average  cost  overrun  of  programs  within  EMD  for  the  period  1980  to  1996  was  40  per¬ 
cent;  the  average  schedule  overrun  was  62  percent.  The  median  cost  overrun  for  the  same 
period  was  20  percent;  the  median  schedule  overrun  was  46  percent.  The  average  performance 
success  rating  was  3.72  out  of  5.  The  average  duration  of  EMD  for  programs  between  1980 
and  1996  was  7.7  years.  Eigures  are  available  by  year  groups  and  by  Service. 

2.  Schedule  slips,  particularly  of  early  test  events,  become  the  best  early  indicators  of  trouble. 

3 .  Eor  programs  that  must  submit  an  exception  S  AR  before  they  are  through  50  percent  of  their  actual 
EMD  duration,  it  is  most  probable  they  will  have  a  schedule  overrun  of  more  than  two  years. 

4.  Using  only  Research,  Development,  Test  and  Engineering  (RDT&E)  costs  during  EMD  pro¬ 
vides  a  comparable  and  possibly  better  analysis  of  the  true  EMD  cost  growth  percentage  than 
using  the  combined  RDT&E  and  Procurement  costs  when  interested  in  managerial  effectiveness. 

5.  The  complexity  of  these  acquisition  programs  makes  it  difficult  to  attribute  a  particular  out¬ 
come  to  any  set  of  two  (or  even  three)  program  characteristics. 

6.  The  OSD/Office  of  Acquisition  Resources  and  Analysis  maintains  a  single  trend  metric,  “Major 
Defense  Acquisition  Program  Cost  Growth.”  This  is  a  required  macro-economic  indicator  but 
does  not  reflect  trends  in  program  execution. 

7 .  The  DSMC  Research  project  is  the  only  system  of  metrics  that  uniquely  combines  performance 
data  (operational  test  results)  with  general  management  data  (cost  and  schedule)  within  EMD. 

Recommendations 

1 .  The  quarterly  Defense  Acquisition  Executive  Summary  process  and  report  is  the  Department 
of  Defense  (DoD)  method  to  address  potential  problems  (early  warning).  It  appears  to  be 
working  well  and  it  should  continue  to  be  used  by  the  top  acquisition  managers  within  DoD. 

2.  There  appears  to  be  a  lack  of  sufficient  trend  data  available  in  DoD  to  determine  if  acquisition 
management  is  improving.  This  research  project,  or  equivalent,  could  be  used  as  an  additional 
trend  metric  and  should  be  continued. 
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3 .  Next  year  is  the  first  year  sufficient  programs  that  have  benefited  from  the  Acquisition  Reform 
(AR)  initiatives  will  have  completed  EMD  and  their  EMD  success  can  be  compared  to  pre-AR 
programs  using  the  data  in  the  DSMC  spreadsheet. 

4.  The  public  perceives  DoD  acquisition  as  inefficient.  However,  there  is  data  showing  that  DoD 
acquisition  is  relatively  good  when  compared  to  other  public  and  private  major  developments. 
Since  perception  is  as  important  as  reality,  it  is  recommended  that  the  relative  efficiency  of 
DoD  acquisition  be  advertised. 

5 .  The  majority  of  programs  have  schedule  overruns  of  less  than  85  percent  according  to  schedule/ 
overrun  data,  while  fewer  than  a  third  of  the  programs  had  schedule  overruns  of  more  than  125 
percent.  This  appears  to  represent  a  significant  difference  between  the  two  groups.  Eurther 
analyses  should  consider  treating  these  groups  separately. 

6.  Multi-variant  analysis  of  program  outcomes  should  be  performed,  to  isolate  significant  factors 
or  combinations  of  factors  in  success  or  failure. 
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1.  Background 


In  1994  the  Science  Advisor  to  the  DOT&E,  or  Director,  Operational  Test  and  Evaluation, 
asked  the  Defense  Systems  Management  College  Test  and  Evaluation  Department  (DSMC  T&E)  to 
research  the  relationship,  if  any,  between  the  number  of  test  articles  used  in  the  Engineering  and 
Manufacturing  Development  (EMD)  phase  of  a  program  acquisition,  and  the  success  of  that  pro¬ 
gram  in  EMD.  Since  determining  the  number  of  test  articles  required  is  a  difficult  trade-off  among 
several  factors,  the  DOT&E  wanted  to  know  if  data  was  available  that  would  help  in  making  this 
important  decision.  Indications  were  that  factual  metrics  relating  to  the  subject  were  nonexistent. 

Unless  specifically  detailed  in  this  report,  all  metrics  and  author’s  conclusions  are  detailed  in 
DSMC  Press  Technical  Report  TR  1-99.  (See  Bibliography.)  Eigure  1,  also  contained  in  TR  1-99, 
shows  the  average  cost  and  schedule  overruns  as  ratios  of  planned  to  actual  figures.  To  obtain 
overrun  percentages,  subtract  1  from  the  values  on  the  axes.  The  average  actual  duration  of  EMD 
for  any  program,  for  any  year  in  which  that  program’s  EMD  phase  ended,  can  be  found  in  the 
database  section  of  TR  1-99. 
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Figure  1.  Ratio  of  Actual  to  Planned  EMD  Cost  and  Schedule 
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2.  DoD  Relative  Cost  Growth 


Figure  2  indicates  the  cost  growth  (overruns)  of  Department  of  Defense  (DoD)  major  weapon 
systems  compared  to  other  complex  public  and  private  sector  projects.  It  shows  that  DoD  cost  over¬ 
runs  are  relatively  small  when  compared  to  the  majority  of  the  other  projects  shown.  Although  Figure 
2  represents  1985,  the  close  similarity  of  the  38  percent  cost  growth  of  1985  when  compared  to  the  40 
percent  cost  growth  of  this  research  suggests  the  data  is  still  valid.  An  updated  Figure  2  every  ten  years 
or  so  might  prove  interesting  and  valuable.  The  database  of  the  current  research  could  be  used  to 
provide  the  DoD  with  updated  cost  data.  A  similar  chart  could  be  created  for  schedule  slips. 

3.  Leading  Indicators 

Recently  the  research  team  was  asked  to  identify  leading  indicators  of  a  program’s  success  or 
problems  within  EMD,  by  a  review  of  the  research  data  available.  This  would  consist  of  data  ob¬ 
tained  from  the  Selected  Acquisition  Reports  (SAR),  the  Operational  Testing  (OT)  reports,  and  the 
DOT&E  independent  evaluation  of  the  Service  OT  reports,  together  with  the  sum  of  the  data  con¬ 
tained  in  the  project  spreadsheet. 

Neither  the  Service  OT  report  nor  the  DOT&E  evaluation  can  provide  leading  indicators  be¬ 
cause  they  occur  near  the  end  of  the  EMD  phase.  Eikewise  the  data  within  the  database  is  historical 
and  for  any  program  the  data  is  not  entered  until  the  program  completes  EMD. 

The  SAR  report  is  a  thoroughly  reviewed,  legally  required  annual  program  report  (quarterly  by 
exception)  that  is  sent  to  Congress.  Its  primary  purpose  is  to  be  the  historically  correct  record  from 
program  inception  (Milestone  (M/S)  I)  through  90  percent  delivered  or  expended.  The  SARs  have 
been  analyzed  several  different  ways,  and  two  potential  early  indicators  were  identified.  (See  pages 
4  and  9.) 

4.  Cost,  Schedule  and  Performance  as  Early  Warnings 

Cost  and  schedule  problems  (overruns)  are  usually  the  effect  of  some  other  program  perturba¬ 
tion,  most  often  shortfalls  in  performance.  Performance,  then,  would  be  the  best  place  to  look  for 
indications  that  a  program  is  in  trouble. 

Unfortunately,  there  are  difficulties  obtaining  performance  data  with  acceptable  confidence  early 
in  the  program,  due  to  limited  numbers  of  test  trials  (see  page  10).  Also,  within  the  program  docu¬ 
mentation  (Acquisition  Program  Baseline  (APB),  SAR,  and  the  Test  and  Evaluation  Master  Plan 
(TEMP)),  performance  is  listed  by  threshold  value,  objective  value,  current  estimated  value,  and 
demonstrated  value  to  date.  In  the  early  stages  of  EMD  it  is  perfectly  normal  for  the  demonstrated 
performance  to  be  below  the  final  threshold  requirement.  But  with  EMD  durations  averaging  8 
years  or  more,  it  is  not  reasonable  to  expect  a  program  manager  to  say  early  on  that  the  eventual 
threshold  requirement  will  not  be  met. 

Cost  status  is  not  a  good  place  to  look  for  early  indicators  of  trouble.  As  mentioned,  cost  growth 
is  usually  a  consequence  of  some  other  occurrence  and  not  a  lead  event.  Also,  cost  status  lends  itself 
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more  to  manipulation  than  does  schedule  or  performance.  For  example,  the  program  can  be  descoped 
by  reducing  logistics  support  or  test  programs.  Activities  can  also  be  postponed  into  the  production 
phase.  The  research  did,  however,  uncover  some  interesting  aspects  of  program  costs  and  cost 
reporting.  (See  page  8.) 

Schedule  status,  then,  becomes  the  best  early  indicator  of  trouble.  To  provide  early  indication, 
the  milestones  section  of  the  APB,  SAR,  DAES,  and  TEMP  must  contain  sufficient  early  test  start 
and  stop  milestones.  Given  sufficient  detailed  milestones  in  program  documentation,  a  slip  in 
early  test  milestones  is  the  best  indicator  of  a  program  heading  into  trouble. 

5.  Sole  Trend  Metric 

The  research  team  believes  the  Office  of  the  Secretary  of  Defense  (OSD)  office  most  involved 
with  the  analysis  of  the  status  of  DoD  acquisition  management  is  the  OSD/AR&A  (Office  of  Acqui¬ 
sition  Resources  and  Analysis).  That  office  provides  OSD  involvement  and  oversight  into  the  APB, 
DAES  and  SAR  procedures,  as  well  as  other  responsibilities.  Eigures  3  and  4  describe  the  sole  trend 
metric  maintained  by  the  office.  This  is  a  macro-economic  metric  designed  to  record  and  control 
overall  funding  levels  for  Major  Defense  Acquisition  Program  (MDAP),  but  does  not  reflect  trends 
in  program  execution. 

To  better  gauge  improvements  in  DoD  acquisition  management  over  time,  we  recommend  that 
the  DAES  process  get  extensive  use,  that  detailed  test  milestones  be  required  within  SARs  and  other 
program  documentation,  and  that  additional  trend  metrics  be  used  (such  as  the  DSMC  EMD  Perfor¬ 
mance  Trends  Research,  or  equivalent). 
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Figure  4.  MDAP  Cost  Growth  Metric  ■  Resuits  for  Past  6  Years 
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Navy  +1.5 
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%  Change  Primary  Drivers  +/-% 

ATIRCM  (+23.7),  BAT  (+21.8), 
FMTV  (+17.4),  LB  APACHE  (+9.7) 
F/A-18  E/F  (+4.1),  JSF  (+6.5), 
JSOW  (-24.3),  SSN  774  (+4.4) 
C-17A  (+2.2),  NAVSTAR  (+8.7), 
SBIRS  (+19.4),  TITAN  IV  (-5.1) 
NAVY  AREA  (+13.0+,  NMD  (+43.4), 
PAC-3  (+17.9),  THAAD  (+22.4) 


6.  Mean  vs.  Median 


The  mean  cost  (n=33)  and  schedule  (n=34)  overruns  for  programs  whose  EMD  phase  ended 
between  1980  and  1996  were  40  percent  in  cost  and  62  percent  in  schedule.  The  median  overruns 
were  20  percent  in  cost  and  46  percent  in  schedule. 

Cost  overrun.  The  large  difference  between  mean  and  median  mostly  reflects  the  presence  in  the 
data  set  of  two  extreme  values:  309  percent  for  Joint  Tactical  Information  Distribution  System 
(JTIDS)  and  201  percent  for  Joint  Surveillance  Target  Attack  Radar  System  (JSTARS).  These  two 
alone  account  for  a  9  percent  increase  in  the  mean  due  to  the  presence  of  JTIDS  [(309-31)732]  and 
a  5  percent  increase  in  the  mean  due  to  the  presence  of  JSTARS  [(210-35)732].  One  commonly  used 
method  to  describe  a  data  set  without  the  skewing  due  to  one  or  two  extreme  numbers  is  to  use  the 
median.  In  the  case  of  cost  overruns,  the  median  of  20  percent  better  describes  the  general  set  of 
programs  than  does  the  mean.  This  is  clearly  demonstrated  by  the  mean  of  3 1  programs  (26  percent) 
and  mean  of  30  programs  (23  percent),  which  shows  the  means  of  the  more  homogeneous  popula¬ 
tions  approaching  the  median  of  the  general  population. 

Schedule  overrun.  The  difference  between  mean  and  median  is  smaller,  62  percent  vs.  46  per¬ 
cent.  This  is  because  there  are  no  extreme  values  in  the  data  set  as  there  are  in  the  cost  overruns.  The 
two  largest  schedule  overruns  were  175  percent  (Maverick)  and  172  percent  (Family  of  Medium 
Tactical  Vehicles).  Together,  these  two  overruns  added  only  7  percent  to  the  mean.  In  fact,  there 
were  four  programs  with  cost  overruns  between  167  percent  and  175  percent:  the  above  two  plus 
Airborne  Self- Protected  Jammer  (ASPJ)  and  Tomahawk.  In  this  case,  it  can  be  argued  that  four  such 
figures,  representing  more  than  10  percent  of  the  programs,  are  too  many  to  be  non-representative 
of  the  data  set.  In  such  a  case,  the  mean  is  a  valid  statistic  to  describe  the  set. 

There  is  at  least  one  other  descriptive  possibility  for  this  type  of  data  set:  the  possibility  of  a  bi- 
modal  or  even  tri-modal  population.  This  involves  examining  the  data  to  see  if  there  are  two  or  three 
separate  groups  within  the  set.  If  there  are,  analysis  would  have  to  explore  possible  reasons  for  such 
groupings  among  the  development  programs.  Schedule  overruns  appear  to  form  at  least  three  popu¬ 
lations:  -20  percent  to  40  percent  (n=15),  40  percent  to  80  percent  (n=10),  and  125  percent  to  175 
percent  (n=9).  The  first  two  may  form  the  same  population. 

The  nature  of  bi-modal  populations  is  that  any  mean  taken  for  the  whole  group  is  irrelevant; 
each  sub-group  has  to  be  analyzed  separately.  In  fact,  for  schedule  overrun,  it  may  be  productive 
to  look  at  just  two  groups:  <80  percent  (n=26)[mean=32,  median=31.5]  and  >125  percent 
(n=9)[mean=149,  median=136].  Of  course,  it  may  be  productive  to  examine  in  what  ways  the 
groups  differ,  other  than  in  the  primary  variable  of  schedule  overrun.  In  the  case  of  schedule 
overrun,  the  >125  percent  group  consists  of  JTIDS,  Maverick,  Tomahawk,  Advanced  Medium 
Range  Air  to  Air  Missile  (AMRAAM),  B-IB,  ASPJ,  Mk48  Advanced  Capability  (ADCAP),  and 
JSTARS.  The  significant  analytical  question  is  whether  these  programs  had  anything  in  common, 
which  was  not  present  in  the  other  programs. 

7.  The  DAES  Process  and  Report 

The  Defense  Acquisition  Executive  Summary  (DAES)  process  and  report  are  recommended  as 
the  principal  method  for  effective  oversight  of  DoD  acquisition  management  and  are  briefly  de¬ 
scribed  on  Figures  5  and  6. 
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•  management  process', 

•  Based  on  quarterly  reports  from  Program 
Managers; 

•  Addresses: 

♦  Program  execution  status  against  the  Acquisition 
Program  Baseiine; 

-  Cost,  Schedule,  and  Performance  Goals. 

♦  Potential  problems  (early  warning). 


Source:  AR&A  Briefing  at  DSMC. 


Figure  5.  Defense  Acquisition  Executive  Summary  (DAES) 
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forward  to  OSD 

Source;  AR&A  Briefing  at  DSMC. 

OSD  Evaluates 
DAES  and 
holds  meeting 

Figure  6.  DAES  Process 
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8.  Aspects  of  EMD  Costs 


Consistent  with  the  idea  that  all  data  should  be  that  which  are  available  to  OSD-level  decision 
makers,  cost  data  for  all  systems  were  collected  from  the  system  SARs.  As  the  period  of  interest  was 
EMD,  the  Research,  Development,  Test  and  Evaluation  (RDT&E)  and  Procurement  costs  were 
collected  for  the  period  from  M/S  II  through  M/S  III.  Planned  costs  were  those  reported  in  the  S  AR 
following  M/S  II;  actual  costs  were  those  reported  in  the  SAR  following  M/S  III. 

In  theory,  one  could  go  to  the  program  SAR  and  retrieve,  rather  directly,  the  planned  and  actual 
costs  of  EMD.  In  practice,  the  retrieval  of  costs  from  the  SAR  is  not  simple.  Eor  one  thing,  costs  are 
reported  in  several  places  in  the  SAR:  for  example,  sections  11,  13,  14,  and  16.  Eor  another,  costs 
are  reported  in  both  Base  Year  and  Then  Year  dollars;  the  inflators  for  each  appropriation  are  not 
always  explicit.  And  yet  another,  the  costs  are  reported  by  fiscal  year.  EMD  need  not  start  or  end  in 
conjunction  with  fiscal  years.  This  leaves  the  analyst  with  the  problem  of  determining  how  much  of 
a  particular  fiscal  year’s  costs  should  be  allocated  to  EMD  for  the  spreadsheet. 

Base  Year  vs.  Then  Year.  In  principle,  the  use  of  Base  Year  figures  will  remove  the  effect  of 
inflation  on  the  figures,  making  it  easier  to  perform  analyses.  In  the  case  of  the  EMD  spreadsheet, 
the  costs  are  used  as  a  ratio  of  planned  to  actual.  Eor  most  programs,  the  same  inflators  will  be  used 
in  both  sets  of  figures,  thereby  yielding  the  same  results  from  either  Base  or  Then  Year  figures.  Eor 
those  programs  in  which  actual  EMD  fiscal  years  were  significantly  longer  than  planned,  when 
inflation  might  have  had  an  effect  on  the  ratios,  use  of  either  figure  will  still  identify  a  significant 
overrun.  In  addition,  several  programs  changed  their  Base  Year  for  SAR  reporting  during  EMD. 
This  creates  additional  difficulties  in  using  Base  Year  figures  for  the  spreadsheet.  Eor  all  the  above 
reasons,  this  analysis  used  Then  Year  costs. 

SAR  Sections.  Sections  11,  13  and  14  do  not  separate  costs  by  fiscal  year.  It  is  not  possible  to 
determine  costs  for  EMD  alone  from  these  figures.  Section  16  is  the  sole  portion  of  the  SAR  that 
records  costs  by  fiscal  year.  Start  of  EMD  (M/S  II)  found  in  section  9  determines  the  earliest  fiscal 
year  to  use.  If  M/S  II  is  in  the  middle  of  a  fiscal  year,  costs  can  be  pro-rated  month  for  month.  M/S 
III  determines  the  end  of  EMD.  If  M/S  III  is  in  the  middle  of  a  fiscal  year,  the  end  of  Initial  Opera¬ 
tional  Test  and  Evaluation  (lOT&E)  can  be  used  to  determine  the  end  of  EMD  expenditures.  With 
the  above  adjustments,  section  16  figures  were  used  for  the  SAR  cost  data. 

RDT&E  Costs.  Eor  some  programs  it  is  difficult  to  determine  what  RDT&E  costs  during  the 
period  of  EMD  actually  were  incurred  in  development  of  the  program.  In  particular,  when  the  program 
includes  several  models,  or  includes  preplanned  product  improvements,  the  program  can  incur  RDT&E 
costs  supporting  these  other  projects  during  the  period  that  the  basic  development  is  in  EMD.  An 
excellent  example  of  this  is  the  (JTIDS)  program,  in  which  class  2H/2M  development  was  initiated 
before  class  2  completed  M/S  III.  SAR  data  can  not  isolate  the  costs  of  the  basic  program,  except  in 
those  cases  where  the  program  office  identified  the  parts  of  the  project  separately  in  the  SAR.  Addi¬ 
tionally,  for  programs  with  preplanned  product  improvements,  total  cost  figures  (such  as  are  in  sec¬ 
tions  11, 13  or  14)  will  not  define  the  basic  development.  In  those  instances  where  cost  definition  was 
not  clear,  best  estimates  of  allocation  to  the  basic  development,  year  by  year,  were  made. 
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Procurement  Costs.  The  SARs  do  not  separately  identify  the  cost  of  Low  Rate  Initial  Produc¬ 
tion,  (LRIP).  The  best  approximation  is  to  use  all  procurement  costs  for  those  years  before  M/S  III. 
This  is  predicated  on  the  legal  constraint  against  using  full  rate  production  funds  before  that  deci¬ 
sion.  Procurement  costs  also  have  the  same  separation  problem  as  discussed  for  RDT&E  above.  For 
programs  with  several  models,  the  same  solution  was  used. 

RDT&E  vs.  Procurement.  There  remains  a  significant  problem  in  determining  the  costs  to  be 
used  in  calculating  EMD  cost  overrun.  For  any  program  with  more  than  a  few  ERIP  items,  the 
procurement  costs  in  EMD  will  be  significantly  (in  some  cases  overwhelmingly)  larger  than  RDT&E 
costs.  This  results  in  the  program  being  seored  for  EMD  cost  overrun  predominantly  by  proeure- 
ment  costs.  The  problem  is  that  a  large  number  of  ERIP  items  may  indicate  a  very  bad  program  (they 
need  to  continue  EMD  to  keep  the  line  warm  while  the  program  keeps  trying  to  pass  lOT&E,  for 
example).  A  large  number  of  ERIP  items  may  also  indicate  a  very  good  program  (testing  in  EMD  has 
been  so  successful  the  Services  demand  the  item  in  the  field  before  M/S  III  for  example).  Thus, 
overwhelming  RDT&E  with  Proeurement  costs  produce  equivocal  data.  For  this  reason,  EMD 
costs  for  RDT&E  and  Procurement  were  entered  separately  in  the  spreadsheet,  with  RDT&E  costs 
alone  being  used  for  the  EMD  cost  overrun  calculation.  Analysis  of  procurement  costs  may  still  be 
performed,  if  desired,  since  the  data  is  present  in  the  spreadsheet.  It  should  be  noted  that  a  sensitivity 
analysis  was  performed  on  the  data  from  the  systems  reported  in  1995.  The  result  was  that  the  use  of 
RDT&E  costs  alone  produced  the  same  EMD  cost  success  factor,  within  a  score  of  1,  in  13  out  of 
14  programs. 


9.  Timing  of  First  Exception  SAR 

The  SAR  process  requires  that  an  exception  SAR  be  submitted  to  OSD  within  90  days  after  the 
program  office  becomes  aware  of  either  a  cost  or  schedule  breaeh  of  the  APB.  An  exception  SAR 
must  also  be  provided  if  there  is  a  Nunn-McCurdy  unit  cost  breach.  The  only  performance  breach 
known  to  the  research  team  is  if  a  program  fails  to  meet  its  exit  eriteria.  Meeting  exit  criteria  require¬ 
ments  is  necessary  for  a  program  to  exit  one  stage  of  the  development  program  and  proeeed  into  the 
next  phase. 

The  hypothesis:  if  the  first  exception  SAR  had  to  be  submitted  early  in  the  program,  it  might  be 
used  as  an  early  indicator  of  a  program  having  difficulty.  Fifteen  program  SARs  were  reviewed  to 
determine  when  the  first  exception  SAR  was  submitted  on  each  program.  The  timing  was  expressed 
as  a  percentage  of  total  time  the  program  was  in  EMD.  It  ranged  from  9  percent  through  to  99 
percent.  The  first  exception  SAR  for  many  programs  was  submitted  more  than  50  percent  through 
EMD  and  thus  could  not  be  used  as  an  early  indicator. 

Figure  7  shows  that  for  programs  whose  first  exception  SAR  was  before  the  program  was  50 
percent  through  EMD,  the  eventual  sehedule  success  at  the  end  of  EMD  was  a  low  success  rating  of 
1  for  each  of  the  eight  programs  involved.  Apparently,  submission  of  a  program’s  first  exception 
SAR  before  EMD  is  50  percent  complete  is  an  indicator  of  eventual  schedule  difficulties. 
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•  Schedule  Success  o  Cost  Success  +  Performance  Success 
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Figure  7.  Time  (Percentage)  into  EMD  of  First  Exception  SAR 

Schedule  Success  Rating  Table 


Success  Rating 


Schedule  Overrun 


5 

4 

3 

2 

1 


On  Schedule 
Less  than  12  months 
Less  than  1 8  months 
Less  than  24  months 
More  than  two  years 


10.  Test  Type  Errors 

Hypothesis  testing  is  used  to  determine  whether  or  not  a  statement  about  a  population  perfor- 
manee  should  or  should  not  be  rejected.  That  is,  an  inference  concerning  the  population  perfor- 
manee  is  made  based  on  a  sample  of  that  population.  The  null  hypothesis  (H^)  expresses  an  assump¬ 
tion  that  the  eomponent,  subsystem  or  system  under  test  meets  the  requirement.  The  alternative 
hypothesis  (HJ  is  that  the  component,  subsystem  or  system  doesn’t  meet  the  requirement.  Four 
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possible  outcomes  are  possible.  1)  is  not  rejected  when  the  actual  population  meets  the  require¬ 
ment,  2)  is  rejected  when  the  actual  population  doesn’t  meet  the  requirement,  3)  is  rejected 
when  the  actual  population  meets  the  requirement,  and  4)  is  not  rejected  when  the  actual  popu¬ 
lation  doesn’t  meet  the  requirement.  The  first  two  outcomes  are  correct  conclusions  and  the  latter 
two  represent  errors.  The  first  error  (H^  is  rejected  when  the  actual  population  meets  the  require¬ 
ment)  is  a  Type  I  or  producer’s  error.  The  second  (H^  is  not  rejected  when  the  actual  population 
doesn’t  meet  the  requirement)  is  a  Type  II  or  consumer’s  error. 

Within  the  acquisition  and  operational  communities,  when  a  Type  I  error  occurs  the  developers 
must  take  action  to  correct  deficiencies  to  meet  requirements  that  are  already  met.  When  a  Type  II 
error  occurs,  the  component,  subsystem  or  system  that  doesn’t  meet  requirements  is  advanced  to 
the  next  stage  of  development  or  fielded.  The  deficiencies  are  then  uncovered  when  they  are  more 
expensive  to  fix  or  when  the  user  needs  the  capability.  As  the  risk  of  committing  a  Type  I  error 
increases,  the  risk  of  committing  a  Type  II  error  decreases  and  visa-versa. 

Based  on  Figure  7,  programs  with  Exception  SARs  had  a  noticeably  better  performance  success 
rating  then  cost  or  schedule  ratings.  This  indicates  a  proclivity  to  ensure  system  performance  meets 
requirements  before  fielding  the  system.  Almost  all  of  these  systems  entered  and  finished  lOT&E 
later  then  planned,  but  the  overall  inferred  performance  indicates  operationally  effective  and  suitable 
systems. 

These  trends  suggest  a  historical  development  process  and  operational  test  and  evaluation  com¬ 
munity  willing  to  accept  increased  producer’s  error  to  reduce  the  risk  of  fielding  systems  that  don’t 
meet  operational  requirements. 

11.  Review  of  the  Technical  Section  of  the  SAR 

The  technical  section  of  the  SARs  during  EMD  for  the  JSTARS  and  AMRAAM  were  reviewed 
to  determine  if  any  leading  indicators  of  program  success  or  trouble  could  be  found.  This  involved 
reading  nine  JSTAR  and  six  AMRAAM  SARs  rendered  during  EMD. 

Eor  JSTARS,  the  technical  section  started  with  6  technical  and  6  operational  parameters.  These 
were  increased  in  number  and  later  decreased  in  number.  Many  explanatory  notes  were  necessarily 
included  to  define  the  conditions  of  evaluating  parameters.  Demonstrated  values  appeared  late  in 
EMD,  with  many  parameters  to  be  determined  (TBD). 

Eor  AMRAAM,  there  were  7  operational  characteristics  and  technical  characteristics  in  the 
areas  of  weight,  length.  Reliability  and  Maintainability,  and  Built-in-Test.  There  was  a  robust  listing 
of  technical  criteria  and  good  notes.  There  were  very  few  test  milestones  listed. 

The  conclusion:  it  is  not  possible  to  obtain  leading  indicators  of  a  program  problem  in  the  tech¬ 
nical  section  of  the  SAR.  Most  Acquisition  Category  (ACAT)  I  programs  are  so  complex  that  not 
even  a  very  good  engineer  analyst,  not  in  the  program  office,  could  follow  the  technical  evolution  of 
the  system. 
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Interestingly,  in  the  JSTAR  program,  where  there  were  good,  detailed  test  milestones,  the  first 
indication  of  program  difficulties  was  a  slip  of  Air  Force  and  Army  test  start  dates  mentioned  in  the 
second  program  EMD  SAR. 

12.  Acquisition  Reform  Impact 

Figure  8  shows  that  there  have  been  4  SAR  programs  which  have  completed  EMD  and  which 
had  most  of  their  EMD  phase  after  implementation  of  Acquisition  Reform  measures.  These  programs 
were  Army  Tactical  Missile  Systems  (ATACMS)  Block  1 A  (Anti-Personnel  Anti-Materiel  (APAM)), 


Joint  Standoff  Weapon  (JSOW),  Secure  Mobile  Antijam  Reliable 
and  Minuteman  (MM)  III  Guidance  Replacement  Program  (GRP). 

Tactical  Terminal  (SMART-T) 

Program 

Cost 

Schedule 

Performance  tOSD  Overall) 

Success/%  overrun 

Success/ %  overrun 

Success 

ATACMS 

4 

5 

3 

16 

4.3 

JSOW 

4 

10 

4 

3 

3.3 

SMART-T 

4 

27 

4 

4 

1.3 

MMIII  GRP 

3 

43 

1 

31 

4 

Mean 

3.75 

21 

3 

34 

3.2 

Median  * 

4 

19 

3.5 

10 

3.65 

For  comparison 

All  programs  1980-1996 

Mean 

3.3 

40 

2.5 

62 

3.8 

Median  * 

20 

46 

What  is  striking  about  the  post-AR  group  is  the  excellent  Schedule  success.  However,  in  looking 
deeper  into  this  result,  another  factor  may  be  significant.  Two  of  the  four  post-AR  programs  were 
Mods.  Modification  programs  can  be  expected  to  have  better  cost  and  schedule  results  than  new 
development  programs,  and  the  data  demonstrate  this. 

All  programs  1980-1996 
Mods 

Mean  3  45 

Median  *  g  4 

Non-Mods 

Mean  49  68 

Median  *  35  52 

Therefore,  the  excellent  results  for  post-AR  programs  might  simply  be  because  half  of  them  were 
modification  programs.  Examination  of  programs  that  have  completed  EMD  under  AR  policies,  to 
determine  the  effectiveness  of  those  policies,  must  wait  for  more  data,  particularly  a  larger  study 
population  of  such  programs. 

*Eor  a  comparison  of  the  use  of  medians  rather  than  means  to  describe  these  populations,  see  page 
6. 
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Figure  8.  Effect  of  Acquisition  Reform  on  Program  EMD 


13.  Use  of  OA  during  EMD 


Programs  within  the  database  were  reviewed  to  determine  if  a  program  used  an  OA  and  if  that 
had  any  effect  on  the  cost,  schedule,  or  performance  success  of  that  program.  Performance  success 
is  the  DOT&E  overall  success  rating  for  the  program. 


Program  No. 

Cost 

OA  USED 

Success  Rating 

Schedule 

Performance 

11 

5 

5 

4 

37 

5 

2 

5 

30 

4 

5 

3 

18 

5 

1 

5 

20 

5 

2 

2 

13 

4 

1 

1 

36 

3 

1 

5 

17 

2 

1 

3 

41 

1 

1 

2 

n  =  9  Average 

3.8 

2.1 

3.3 

Program  No. 

Cost 

OA  NOT  USED 

Success  Rating 

Schedule 

Performance 

1 

3 

4 

5 

5 

4 

4 

4 

9 

5 

4 

3 

40 

5 

1 

5 

7 

4 

1 

5 

19 

4 

1 

5 

31 

4 

1 

4 

10 

1 

1 

4 

22 

3 

1 

2 

3 

1 

2 

1 

23 

1 

1 

J_ 

n  =  1 1  Average 

3.2 

1.9 

3.5 

Initial  Conclusions 

The  use  of  OAs  had  a  positive  result  for  cost  and  schedule  success  and  a  negative  result  for 
performance.  Schedule  and  performance  success  differences  were  close  to  each  other.  Cost  success 
using  OAs  seems  to  have  made  the  greatest  difference.  In  all  cases  the  data  is  broadly  scattered. 

Since  these  results  are  counter-intuitive,  more  analysis  and  more  data  are  required. 

14.  Program  Characterization 

During  the  research  effort  it  was  necessary  to  read  every  SAR  written  on  a  program  during 
EMD.  This  being  so,  it  was  thought  that  a  short  one-  or  two-page  summary  of  the  program’s  ups  and 
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downs  as  recorded  in  the  SAR  would  complement  the  statistical  data  gathered  on  the  program  and 
entered  into  the  spreadsheet.  Perhaps  the  qualitative  and  quantitative  data  together  would  provide 
early  indicators  or  “useful  lessons  learned”  data. 

When  we  considered  developing  an  advanced  relational  database,  we  thought  this  written  pro¬ 
gram  description  could  contain  a  key  word/phrase  component  that  would  allow  searching  the  data¬ 
base  by  single  element  or  a  combination  of  a  few  elements/key  words.  The  attached  C-17  program 
written  characterization  is  an  example. 

However,  it  was  determined  the  added  effort,  particularly  without  the  advanced  database,  was 
not  warranted. 


C-17  Program  Characterization 


Profile 


HMD  Duration:  10  yrs  9  mos  M/S  II  date:  Feb.  85 

EMD  Cost  Overrun:  38  percent  M/S  III  date:  Nov.  95 

EMD  Schedule  Overrun:  79  percent 

Performance  Success  Ratings  AFOTEC  DOT&E 

Operational  Effectiveness:  5  5 

Operational  Suitability:  5  5 

Overall  Performance:  5  5 

(  5  is  very  good,  max.  down  to  1,  poor,  min.) 


Technical  Aspects 

The  C-17  technical  development  effort  was  singularly  successful.  At  the  beginning  of  EMD  the 
required  technical  characteristics  were  limited  to  two  technical  and  five  operational;  all  related  to 
major  required  characteristics  of  large  transport  aircraft.  Two  years  later  the  USD(A&T)  directed 
the  inclusion  of  four  additional  requirements,  and  five  years  into  EMD  two  additional  requirements 
were  added.  Nine  years  into  EMD  an  additional  four  requirements  were  added,  totaling  17  critical 
requirements.  However,  the  final  requirements  defined  in  the  ORD  and  validated  by  the  Joint  Re¬ 
quirements  Oversight  Council  (JROC)  were  only  seven,  again  all  related  to  the  major  intended  use 
of  this  cargo  aircraft,  including  paratrooper  drops. 

The  major  problems  encountered  during  technical  development  included  a  static  load  wing  fail¬ 
ure,  and  a  case  of  two  paratroopers  making  contact  during  a  dual  door  airdrop.  Both  problems  were 
solved  satisfactorily.  The  Eive  Eire  Test  requirements  of  the  law  and  imposed  by  DOT&E  was  a 
protracted  and  contentious  issue.  However,  in  the  SAR  reporting,  the  subject  is  treated  in  a  professional 
matter-of-fact  manner.  The  addition  of  a  Defensive  Avionics  Capability  to  the  aircraft  and  a  rigorous 
Eive  Eire  Test  and  Evaluation  (EET&E)  program  was  successfully  completed. 

Initially  engineering  manpower  was  in  short  supply,  impacting  design  changes,  and  overall  cost 
and  schedule.  The  Program  Management  Office  (PMO)  authorized  several  technical  initiatives:  Product 
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Variability  Reduction,  Product  Engineering,  and  Value  Engineering.  And  in  the  last  year  of  EMD  a 
Product  Enhancement/Performance  Improvement  task  was  awarded  to  “mature  the  aircraft.” 

The  overall  technical  design  effort  resulted  in  an  aircraft  that  closely  met  all  its  original  require¬ 
ments.  Although  a  few  minor  limitations  were  noted  in  the  lOT&E  reports,  both  reports  were  ex¬ 
tremely  positive.  The  design  team  has  also  been  awarded  several  prestigious  industry  awards  for 
their  efforts. 

Business  Aspects 

The  business  aspects  of  the  C-17  program  were  more  tumultuous  than  the  technical.  The  initial 
EMD  annual  SAR  report  talks  of  Government  reviews  rating  the 

Contractor  “best  yet”  of  any  aircraft  manufacturer,  and  “ready  for  production”  in  a  later  SAR. 
These  initial  reports  quickly  changed  as  cost,  schedule  and  quality  problems  began  to  surface. 
These,  in  turn,  generated  considerable  outside  interest  and  direction  for  the  program  office.  In¬ 
volved  groups  included  Congressional  direction.  Secretary  of  Defense  direction,  a  Joint  DoD 
Cost/Schedule  Control  Systems  Criteria  (C/SCSC)  Surveillance  Review,  and  establishment  of  a 
Senior  Management  Review  Group  in  OSD,  and  advice  and  consultation  by  the  Defense  Science 
Board  (DSB).  Extensive  Contractor  reorganization  and  management  disruptions  exacerbated  the 
program’s  problems. 

Some  directed  program  requirements  resulting  from  outside  interests  included  changes  to  the 
way  costs  were  being  recorded,  cuts  in  EY90  and  91  procurement  funds,  a  90  percent  reduction  in 
planned  procurement  quantity,  and  restrictions  on  the  spending  of  procurement  funds  until  certain 
events  occurred  on  the  production  line  and  certain  operational  assessments  were  completed.  By  far 
the  most  serious  business  impact  upon  the  program  was  the  1993  Defense  Acquisition  Board  (DAB) 
decision  to  only  commit  to  buy  40  aircraft  (A/C)  prior  to  the  M/S  IIIB  Eull  Rate  Production  decision 
scheduled  for  November  1995.  A  40  A/C  buy  was  considered  adequate  to  evaluate  whether  cost, 
schedule  and  performance  of  the  C-17  warranted  the  full  buy  of  120.  Concurrently,  the  DoD  initi¬ 
ated  a  study  of  alternative  A/C  procurement  buys  to  fulfill  the  mission.  Integrated  Product  Teams 
(IPTs)  were  also  initiated  at  this  time. 

Notwithstanding  this  hectic  business  activity  and  early  program  EMD  problems,  the  C-17  pro¬ 
gram  completed  EMD  in  a  superior  manner.  Beginning  in  1994  the  SARs  were  able  to  report  the 
timeliness  of  A/C  delivery  has  “markedly  improved”  and  quality  rework  was  down  40  percent.  The 
C-17  aircraft  set  at  least  22  world  performance  records  by  this  time,  and  the  Government- Industry 
team  was  given  several  awards.  The  most  recent  award  (1998)  was  the  Malcolm  Baldrige  National 
Quality  Award  to  the  (now)  Boeing  Airlift  and  Tanker  Team,  considered  one  of  the  very  best  in 
achieving  quality  and  world-class  business  performance. 

Analysis 


The  C-17  program  had  its  share  of  technical,  and  in  particular,  management  problems.  Erom  a 
rocky  start  it  emerged  from  EMD  as  an  acknowledged  success.  Possible  reasons  for  this  include  a 
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limited  set  of  stated  teehnical  requirements,  the  impact  of  outside  guidance  and  threat  of  considering 
an  alternate  aircraft  to  fulfill  the  mission,  and  time  to  incorporate  all  these  events  within  EMD,  prior 
to  full  rate  production  and  operational  use.  The  EMD  phase  stretched  from  a  planned  6  years  to  near 
11  actual  years. 

Key  Phrases 

Technical  Requirements  (clarity) 

External  Influences  (beneficial) 

EMD  Duration  (sufficient) 


15.  The  Right  Stuff 

Beginning  in  1990,  under  the  direction  of  Dr.  Owen  Gadeken  at  DSMC,  a  major  research  effort 
was  conducted  by  interviewing  74  program  managers  and  by  surveying  an  additional  356  program 
management  personnel.  The  objective  of  the  research  was  to  identify  the  characteristics  that  make 
excellent  program  managers.  The  results  are  the  eight  characteristics  shown  below.  The  underlined 
were  identified  as  being  of  great  importance. 

Are  strongly  committed  to  a  clear  mission 

Have  a  long  term  and  big  picture  perspective 

Are  both  systematic  and  innovative  thinkers 

Eind  and  empower  the  best  people  for  their  product  teams 

Are  selective  in  their  involvement  in  program  issues 

Eocus  on  external  stakeholders 

Thrive  on  relationships  and  influence 

Proactively  gather  information  and  insist  on  results 

This  research  has  recently  been  compared  with  a  more  recent  “Armstrong  Thesis”  of  1999.  The 
following  chart  shows  the  comparative  values  of  similar  attributes  measured  nine  years  later.  Note 
that  three  of  the  four  highest-ranked  competencies  in  both  research  studies  are  the  same. 

The  above  is  simply  an  example  of  the  excellent  acquisition  research  that  exists.  This  type  data 
combined  with  the  DSMC  EMD  research  project  quantitative  data  should  be  very  useful  in  provid¬ 
ing  a  “template”  against  which  a  program’s  M/S  II  EMD  plan  can  be  compared.  If  the  current  Excel 
spreadsheet  of  the  DSMC  research  was  replaced  by  a  relational  database,  these  comparisons  and 
aggregations  of  positive  attributes  could  be  facilitated. 
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Program  Manager’s  Leadership  Competencies  -  Historical  Comparison 


1990  DSMC  Study  (128  PMs) 

1 .  Sense  of  ownership/mission 

2.  Long-term  Perspective 

3.  Managerial  orientation  (16) 

4.  Political  awareness 

5.  Optimizing  (20) 

6.  Results  orientation 

7.  Innovativeness 

7 .  Systematic  thinking  (14) 

9.  Focus  on  excellence 

10.  Relationship  development 

11.  Action  orientation 

12.  Coaches  others 

Repeat  numbers  =  Tie  in  rankings 


1999  Armstrong  Thesis  (39  PMs) 

1 .  Long-term  perspective 

2.  Innovativeness 

3.  Political  awareness 

4.  Sense  of  ownership/mission 

5.  Relationship  Development 
5.  Action  orientation 

7 .  Strategic  influence  (13) 

8.  Results  orientation 

9.  Focus  on  excellence 

1 0 .  Collaborative  influence  (18) 

1 1 .  Professional  expertise  (21) 

12.  Coaches  others 

( )  =  Rank  in  other  survey 
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